Papers with causal reasoning
The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can explain quantum mechanics and write sophisticated code, yet fail to parse sentences like "The cat that the mouse feared chased meowed" |
| Approach: | They propose a framework to distinguish structural understanding from semantic pattern matching . they use a set of 9,720 comprehension questions on center-embedded sentences . |
| Outcome: | a new framework shows that models lose performance when they abandon structural analysis for semantic associations. |
Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models (2021.acl-short)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) achieve surprising performance on the Choice of Plausible Alternatives (COPA) task. |
| Approach: | They propose to add a regularization loss to the existing COPA models to mitigate the problem of semantic similarity bias by adding a normalization loss. |
| Outcome: | The proposed model improves generalization ability and performs better on a challenging dataset, BCOPA-CE, which has unbiased token distribution and is more difficult for models to distinguish cause and effect. |
CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in foundation language models have shown the efficacy of pre-trained models across diverse QA tasks. |
| Approach: | They propose a multi-task benchmark for evaluating causality-aware language models to unify causal QA research. |
| Outcome: | The proposed model outperforms single-task fine-tuned models on the CALM-Bench tasks. |
ACCESS : A Benchmark for Abstract Causal Event Discovery and Reasoning (2025.naacl-long)
Copied to clipboard
Vy Vo, Lizhen Qu, Tao Feng, Yuncheng Hua, Xiaoxi Kang, Songhai Fan, Tim Dwyer, Lay-Ki Soon, Gholamreza Haffari
| Challenge: | Existing methods for identifying event causality in NLP are limited in their scale and rely on lexical cues. |
| Approach: | They propose a benchmark for identifying abstract causality from a large-scale dataset. |
| Outcome: | The proposed benchmark can be leveraged for enhancing QA reasoning performance in LLMs. |
Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly being applied to causal reasoning tasks. |
| Approach: | They propose a symbolic verification framework that checks whether LLM-generated causal expressions are derivable from a given causal graph using do-calculus and probability theory. |
| Outcome: | The proposed framework can recover correct answers that would otherwise be marked incorrect due to superficial differences. |
Causal Direct Preference Optimization for Language Model Alignment (2026.findings-eacl)
Copied to clipboard
| Challenge: | Empirical evaluations show that CDPO surpasses DPO-based baselines by achieving unbiased fine-tuning through causal reasoning. |
| Approach: | They propose a framework that incorporates causal inference principles to mitigate the influence of confounders and sharpen the signal of genuine human preferences. |
| Outcome: | The proposed framework preserves the tractability of direct optimization while enhancing robustness to spurious correlations and annotation biases. |
ARQA: A Benchmark for Grounded Table–Text QA in Enterprise Annual Reports (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing QA benchmarks focus on retrieval or single-modality reasoning . annual reports are a company's definitive record of performance . |
| Approach: | They propose an annual report QA benchmark that compares QAs with lookups, arithmetics, and insights. |
| Outcome: | The proposed benchmarks show strong factual retrieval but persistent weaknesses in grounded arithmetic and causal reasoning. |
CausalGraph2LLM: Evaluating LLMs for Causal Queries (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have opened up new avenues for their use beyond standard Natural Language Processing tasks. |
| Approach: | They propose a benchmark to evaluate the capabilities of Large Language Models (LLMs) they use over 700k queries to compare their encoding capabilities. |
| Outcome: | The proposed benchmark compared LLMs on graph-level and node-level queries and open-sourced and closed models. |
ExCAR: Event Graph Knowledge Enhanced Explainable Causal Reasoning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work infers the causation between events based on knowledge from annotated causal event pairs, but additional evidence information is unexploited. |
| Approach: | They propose an Event graph knowledge enhanced explainable CAusal Reasoning framework that acquires additional evidence information from a large-scale causal event graph as logical rules for causal reasoning. |
| Outcome: | The proposed framework outperforms state-of-the-art methods in human evaluation and in animal models. |
CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions (2022.findings-acl)
Copied to clipboard
Tayfun Ates, M. Ateşoğlu, Çağatay Yiğit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, Deniz Yuret
| Challenge: | Existing models with similar physical and causal understanding capabilities are still underdeveloped. |
| Approach: | They propose a video question answering dataset that requires causal reasoning about physical forces and object interactions. |
| Outcome: | The proposed dataset requires causal reasoning about physical forces and object interactions. |
Lost in the Distance: Large Language Models Struggle to Capture Long-Distance Relational Knowledge (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent large language models have demonstrated impressive capabilities in handling long contexts . however, as context length increases, LLMs struggle more with filtering out irrelevant information . |
| Approach: | They propose to use unrelated sentences to capture relational knowledge over long contexts . they find that LLMs can handle edge noise with little impact, but can reason about distant relationships . |
| Outcome: | The proposed model can handle edge noise with little impact, but its ability to reason about distant relationships declines as the noise grows. |
What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks often include a mixture of reasoning questions, making it difficult to truly assess VLMs’ causal reasoning abilities. |
| Approach: | They propose two new benchmarks specifically designed to isolate and rigorously evaluate VLMs’ causal reasoning abilities. |
| Outcome: | The proposed benchmarks show that vision-language models perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. |
Multimodal Causal Reasoning Benchmark: Challenging Multimodal Large Language Models to Discern Causal Links Across Modalities (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing MLLMs lack robustness in multimodal causal reasoning compared to their performance in textual settings. |
| Approach: | They propose a novel multimodal chain-of-thought (CoT) reasoning benchmark that leverages siamese images and text pairs to challenge MLLMs. |
| Outcome: | The proposed benchmark leverages siamese images and text pairs to challenge MLLMs. |
Visual Choice of Plausible Alternatives: An Evaluation of Image-based Commonsense Causal Reasoning (L18-1)
Copied to clipboard
Jinyoung Yeo, Gyeongbok Lee, Gengyu Wang, Seungtaek Choi, Hyunsouk Cho, Reinald Kim Amplayo, Seung-won Hwang
| Challenge: | Existing methods for evaluating plausibility of events are focused on measuring causal dependency between events or actions. |
| Approach: | They propose a task to identify the more plausible alternative with their commonsense causal context. |
| Outcome: | The proposed task is based on a visual COPA dataset with 380 questions and over 1K images with various topics. |
Conditional Dichotomy Quantification via Geometric Embedding (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods that rely on semantic similarity fail to capture the nuanced oppositional dynamics essential for these applications. |
| Approach: | They propose a task that formalizes the measurement of conditional dichotomy by using a dichotomian framework. |
| Outcome: | The proposed framework provides carefully constructed datasets covering debate, defeasible inference, and causal reasoning scenarios. |
Exploring Defeasibility in Causal Reasoning (2024.findings-acl)
Copied to clipboard
Shaobo Cui, Lazar Milikic, Yiyang Feng, Mete Ismayilzada, Debjit Paul, Antoine Bosselut, Boi Faltings
| Challenge: | Existing studies ignore defeasibility in causal reasoning and fail to evaluate existing causal strength metrics in defensible settings. |
| Approach: | They propose a metric that measures causal strength based on token-level causal relationships. |
| Outcome: | The proposed metric improves on existing metrics by 69.7% . supporters and defeaters are more effective than opponents, the authors show . |
Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data (2024.findings-acl)
Copied to clipboard
| Challenge: | Quantitative reasoning with data is a critical skill to analyze data, yet the assessment of such ability remains limited. |
| Approach: | They propose a quantitative reasoning with data benchmark to evaluate Large Language Models' ability in statistical and causal reasoning with real-world data. |
| Outcome: | The proposed model GPT-4 achieves an accuracy of 58%, while open-source model Deepseek-coder-instruct gets the highest accuracy of 37%. |
Causal Reasoning through Two Cognition Layers for Improving Generalization in Visual Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing attempts to generalize VQA focus on unimodal aspects, overlooking enhancements in multimodal aspects. |
| Approach: | They propose to decompose the responsibility of each stage into distinct experts and a cognition-enabled component (CC) they prioritize answer predictions governed by pathways involving both CCs while disregarding answers produced by either CC. |
| Outcome: | The proposed model improves multimodal predictions by emphasizing causal reasoning factors. |
The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code (2023.findings-acl)
Copied to clipboard
| Challenge: | entailment a) |
| Approach: | entailment : We want to explore whether Code-LLMs with code prompts are better . encoding a code prompt is better than text-only LLMs, they say . |
| Outcome: | entailment : Our results show that Code-LLMs with code prompts are better compared to text-only LLMs. |
RoboFailRing: Retrieval-Augmented and Language Grounding Failure Detection for VLM-enabled Robotic Manipulation (2026.acl-long)
Copied to clipboard
| Challenge: | RoboFailRing enables timely failure detection during task execution and enhances reasoning accuracy of VLMs. |
| Approach: | They propose a robot-based failure detection system that enables timely failure detection . they evaluate a large-scale simulated dataset and provide a grounded failure report . |
| Outcome: | The proposed method achieves rapid failure detection and returns similarity-based decision on large-scale simulated failures. |
CausalEval: Towards Better Causal Reasoning in Language Models (2025.naacl-long)
Copied to clipboard
Longxuan Yu, Delin Chen, Siheng Xiong, Qingyang Wu, Dawei Li, Zhikai Chen, Xiaoze Liu, Liangming Pan
| Challenge: | Large language models (LLMs) have been used for a variety of tasks, including problem-solving, decision-making, and understanding of the world. |
| Approach: | They propose a review of existing methods aimed at enhancing LMs for causal reasoning . they categorize existing methods as reasoning engines or as helpers providing knowledge or data to traditional methods . |
| Outcome: | The proposed methods perform better than existing methods on a range of tasks. |
Dr.ECI: Infusing Large Language Models with Causal Knowledge for Decomposed Reasoning in Event Causality Identification (2025.coling-main)
Copied to clipboard
| Challenge: | Existing solutions lack generalizability to unseen domains, underscoring the urgent need for generalization capabilities in the field of ECI. |
| Approach: | They propose a multi-agent Decomposed reasoning framework for Event Causality Identification that incorporates specialized agents such as Causal Explorer and Mediator Detector. |
| Outcome: | The proposed framework improves the state-of-the-art performance of LLMs for event causality identification (ECI) tasks compared with baselines based on LLM and supervised training. |
CReTIHC: Designing Causal Reasoning Tasks about Temporal Interventions and Hallucinated Confoundings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated impressive capabilities in natural language processing, but their ability to establish causal relationships remains challenging. |
| Approach: | They propose a novel dataset to test and enhance the causal reasoning abilities of large language models (LLMs) by integrating elements of verbal hallucinations and temporal interventions into existing causal inference datasets. |
| Outcome: | The proposed dataset is designed to test and enhance the causal reasoning abilities of large language models. |
Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate ChatGPT's causal reasoning abilities are based on pre-trained language models, but they rely on supervised training. |
| Approach: | They conduct the first comprehensive evaluation of ChatGPT’s causal reasoning capabilities using four state-of-the-art (STA) simulations. |
| Outcome: | The proposed model is not a good causal reasoner, but a great causal interpreter. |
CausalityCheck: A Framework for Evaluating Causal Reasoning in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods fail to accurately reflect a model's causal reasoning capabilities. |
| Approach: | They propose a tool to automatically generate causal reasoning checklists to assess the causal reasoning abilities of 18 large language models. |
| Outcome: | The proposed tool assesses the causal reasoning abilities of 18 large language models. |
Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need (2023.emnlp-main)
Copied to clipboard
| Challenge: | a scientific claim verification requires thorough examination and assessment to ascertain its validity . attention architectures and pre-trained language models fail to establish a comprehensive chain of causal inference . |
| Approach: | They propose a qualitative causal structure-based graph neural network model to facilitate causal reasoning across relevant causally-potent factors. |
| Outcome: | The proposed model outperforms state-of-the-art models by incorporating semantic features . the proposed model is based on a qualitative causal structure . |
LORAXBENCH: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | LORAXBENCH is a benchmark for low-resource languages of Indonesia . it covers reading comprehension, open domain QA, language inference, causal reasoning, translation, and cultural question answering across 20 languages. |
| Approach: | They propose a benchmark that focuses on low-resource languages of Indonesia and covers 6 diverse tasks: reading comprehension, open-domain QA, language inference, causal reasoning, translation, and cultural question answering. |
| Outcome: | The proposed benchmark covers reading comprehension, open-domain QA, language inference, causal reasoning, translation, and cultural question answering across 20 Indonesian languages. |
CRAB: Assessing the Strength of Causal Relationships Between Real-world Events (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for reasoning about events in narratives do not understand the complexity of the causal relationships of events in the narrative. |
| Approach: | They propose a Causal Reasoning Assessment Benchmark to evaluate causal understanding of events in narratives. |
| Outcome: | The proposed model performs worse when models are derived from complex causal structures than simple linear causal chains. |
MutantPrompt: Prompt Optimization via Mutation Under a Budget on Modest-sized LMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized the way we learn and process information, but identifying optimal prompts remains a challenge for low-resource languages. |
| Approach: | They propose a framework that leverages multi-armed bandit algorithms to efficiently identify optimal prompts tailored to low-resource languages. |
| Outcome: | The proposed framework is able to find optimal prompts for low-resource languages and significantly improves performance across multiple low-level tasks. |
CausalRAG: Integrating Causal Graphs into Retrieval-Augmented Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing RAG frameworks face critical limitations due to text chunking and semantic similarity. |
| Approach: | They propose a framework that incorporates causal graphs into the retrieval process. |
| Outcome: | The proposed framework preserves contextual continuity and improves retrieval precision, leading to more accurate and interpretable responses. |
CogEvolve: A Multimodal Benchmark for Evaluating Relational Reasoning in Semantic Extension (2026.acl-long)
Copied to clipboard
| Challenge: | a gap exists between human embodied logic and machine statistical learning . authors: models internalize statistical patterns or mimic static recognition . |
| Approach: | They propose a cognitive linguistic benchmark to test whether large language models internalize statistical logic or not . they find that models function as "Super-Associators" expert at static recognition yet fail at causal reasoning . |
| Outcome: | The proposed model fails at causal reasoning and has a high-fidelity concept representation but lacks transformational operators essential for true relational understanding. |
Under the Shadow of Babel: How Language Shapes Reasoning in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | linguistic relativity suggests that the structure of language shapes cognitive patterns . large language models internalize the habitual logical structures embedded in different languages, authors say . |
| Approach: | a study introduces a bilingual dataset for causal reasoning in Chinese and English. |
| Outcome: | a new study shows that large language models internalize reasoning biases shaped by language . the model internalizes language-specific preferences and rigidly applies them to atypical inputs, the study shows . |
How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit increasingly sophisticated linguistic capabilities, yet the extent to which these models reflect human-like cognition versus advanced pattern recognition remains an open question. |
| Approach: | They conduct a series of targeted experiments to assess whether LLMs construct semantic representations and pragmatic inferences in a human-like manner. |
| Outcome: | The proposed framework can be used to assess the cognitive and linguistic capabilities of large language models (LLMs). |
UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces (2025.acl-long)
Copied to clipboard
Baining Zhao, Jianjie Fang, Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jinqiang Cui, Xinlei Chen, Yong Li
| Challenge: | Large multimodal models exhibit remarkable intelligence, yet their embodied cognitive abilities during motion in open-ended urban aerial spaces remain to be explored. |
| Approach: | They propose a benchmark to evaluate whether large multimodal models can process continuous first-person visual observations like humans. |
| Outcome: | The proposed model can process first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. |
SAHM: A Benchmark for Arabic Financial and Shari’ah-Compliant Reasoning (2026.acl-long)
Copied to clipboard
Rania Elbadry, Sarfraz Ahmad, Ahmed Heakl, Dani Bouch, Momina Ahsan, Muhra AlMahri, Marwa Elsaid Khalil, Yuxia Wang, Salem Lahlou, Sophia Ananiadou, Veselin Stoyanov, Jimin Huang, Xueqing Peng, Preslav Nakov, Zhuohan Xie
| Challenge: | English financial NLP has progressed rapidly through benchmarks for sentiment, document understanding, and financial question answering. |
| Approach: | They propose a document-grounded benchmark and instruction-tuning dataset for Arabic financial NLP and Shari’ah-compliant reasoning. |
| Outcome: | The proposed dataset contains 14,380 expert-verified instances spanning seven tasks . it includes financial sentiment analysis, extractive summarization, and event–cause reasoning . |
NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on clean, abstract scenarios where causal structure is simple or implicitly assumed. |
| Approach: | They propose a benchmark to evaluate causal reasoning under structured noise. |
| Outcome: | The proposed method outperforms standard prompting and reasoning baselines on NoisyCausal. |
SERE: Structural Example Retrieval for Enhancing LLMs in Event Causality Identification (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated strong performance across various NLP tasks, but their effectiveness in ECI remains limited due to biases in causal reasoning. |
| Approach: | They propose a structural example retrieval framework that leverages LLMs’ few-shot learning capabilities to help LLM models in ECI. |
| Outcome: | The proposed framework leverages LLMs’ few-shot learning capabilities to guide LLM models in causal reasoning, mitigating bias and improving accuracy. |